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Abstract-  Video-based  eye  gaze  detection  systems  are  use¬ 
ful  for  eye-slaved  support  system  for  the  severely  disabled. 
The  pupil  center  in  the  video  image  is  a  focal  point  to  de¬ 
termine  the  eye  gaze.  Recently,  to  improve  the  disadvan¬ 
tages  of  traditional  pupil  detection  methods,  a  pupil  detec¬ 
tion  technique  using  two  light  sources  (LEDs)  and  the  im¬ 
age  difference  method  was  proposed  [1].  In  addition,  for 
users  or  subjects  wearing  corrective  eyeglasses  a  method 
for  eliminating  the  images  of  the  light  sources  reflected  in 
the  glass  lens  was  proposed.  However,  image-processing 
hardware  for  implementing  these  methods  is  rather  ex¬ 
pensive.  I  n  the  present  paper,  the  hardware  construction  is 
replaced  by  a  construction  consisting  of  a  combination  of  a 
conventional  image  grabber  and  a  personal  computer.  An 
algorithm  for  windowing  around  the  pupil  image  with  an 
automatic  thresholding  method  for  pupil  detection  is  pro¬ 
posed.  The  results  show  that  the  algorithm  works  well 
when  the  user  or  the  subject  is  wearing  eyeglasses  and 
under  normal  ambient  lighting  conditions.  The  calculation 
time  is  quick  enough  for  real  time  processing.  These  algo¬ 
rithms  would  contribute  to  consistent  and  reliable  pupil 
detection. 

Keywords-  pupil  detection,  image  processing,  eye-gaze 
detection,  support  system,  the  disabled 

I  .  I  NTRODUCTION 

Development  of  outstanding  eye  gaze  (or  line  of 
sight)  detection  methods  is  desired  for  several  fields, 
such  as  for  man-machine  interfaces  and  for  psycho¬ 
physical  or  behavioral  studies.  It  is  very  useful  espe¬ 
cially  for  many  of  the  physically  disadvantaged,  such 
as  amyotrophic  lateral  sclerosis  patients,  who  cannot 
use  basic  body  movements  accurately  but  do  indeed 
have  eye  movement  ability.  Such  a  user  is  able  to 
control  peripheral  assistant  devices  or  can  communi¬ 
cate  with  others  through  a  pointer  with  the  computer 
presenting  hierarchal  structural  menu  options  on  a 
computer  screen  [2], 

Generally  the  line  of  sight  is  determined  by  the 
relative  position  between  the  pupil  center  and  glint 
(the  corneal  reflection  light  of  the  light  source)  in  the 
image  because  the  glint  does  not  move  as  much  as  the 
pupil  i  mage  [2]-[4],  There  are  two  conventional  pupil 
detection  methods:  bright  eye  [2][4]  and  dark  eye 
methods  [5][6],  Recently,  to  eliminate  the  shortcom¬ 
ings  of  these  conventional  methods,  a  combination  of 


the  two  methods  was  proposed:  a  pupil  detection 
method  using  two  light  sources  and  the  image  differ¬ 
ence  method  [1],  I  n  this  method,  one  light  source  set 
coaxial  with  thecamera  is  switched  on  during  theodd 
fields  of  the  NTSC  video  signal  and  another  one  set 
uncoaxial  with  thecamera  is  switched  on  during  the 
even  fields.  The  difference  images  are  obtained  by 
subtracting  the  even  field  images  from  the  consecu¬ 
tive  odd  field  images.  Since  the  background  almost 
vanishes  in  the  difference  image,  a  threshold  setting 
for  pupil  detection  is  much  easier.  We  have  also  pro¬ 
posed  a  method  that  allows  the  users  to  wear  correc¬ 
tive  eyeglasses.  It  used  a  special  hardware  device  for 
i  mage  processi  ng.  H  owever,  we  di  d  not  descri  be  the 
thresholding  methods  for  eliminating  the  glass  re¬ 
flection  light  (GRL)  and  detecting  the  pupil  image. 

In  general,  the  GRL  tends  to  make  pupil  detec¬ 
tion  unstable.  For  consistent  and  reliable  pupil  de¬ 
tection,  the  most  important  thing  is  to  consistently 
window  around  the  pupil  image  and  to  mask  the 
noise  in  the  background  (including  the  GRL). 

The  present  paper  proposes  an  algorithm  for 
wi  ndowi  ng  around  the  pupi  I  i  mage  on  the  base  of  the 
difference  pupil  detection  method,  which  works  on  a 
relatively  cheap  construction  (a  personal  computer 
and  an  image  grabber).  It  is  shown  that,  in  the  re¬ 
mote  condition  (the  distance  between  the  camera  and 
eye  is  50  -  65  cm)  and  in  normal  ambient  lighting 
conditions,  the  algorithm  works  well  on  the  images 
including  theGRL. 

II .  Methods 

A.  Experimental  setups 

Optical  devices  for  experiment  have  been  de¬ 
scribed  elsewhere  [1],  The  images  of  an  eye  were 
captured  by  the  image  grabber  (brightness  depth:  0- 
255  with  a  resolution  of  640Hx240V  pixels  for  each 
field,  640Hx480V  pixels  for  oneframe. 

B.  Algorithm  of  Microprocessor -Based  Windowing 
Process  in  Each  Frame 

Step  1.  Making  reduced  images  and  histogram  for 
odd  and  even  field  images.  To  shorten  the  time  for 
accessing  the  image  grabber,  we  sampled  each  field 
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Brightness 

Fig.  1.  Sample  images  and  their  brightness  histograms. 


(a)  Odd  image  (b)  Even  image  (c)  Difference  image 


Brightness  Bdbw 

Fig.  2.  The  temporal  window  (TW)  and  the  brightness  histograms  in  TW. 


image  at  every  16  pixels  horizontally  and  every  8 
pixels  vertically.  The  total  sampled  pixel  numbers 
were  1200  (40Hx30V)  for  each  odd  and  even  field, 
respectively  (Fig.  2(a)  and  (b)). 

Step  2.  Removing  saturated  pixels  in  brightness  and 
their  neighboring  pixels  from  odd  image.  A  fraction 
oftheGRL  image  always  saturates  (level  255)  in  the 
i  mage.  H  owever,  there  i s  the  possi  bi  I  ity  that  the  sur- 
roundings  of  the  saturated  GRL  image  are  higher  in 
brightness  than  the  pupil  image.  The  areas  may  be 
mistakenly  judged  as  the  pupil  area.  In  addition,  the 
glint  usually  saturates,  this  may  be  mistakenly 
judged  as  the  GRL. 

All  pixels  in  the  odd  image  are  scanned  horizon¬ 
tally.  Sincethe  glint  does  not  consist  of  morethan 
one  pixel,  thesaturated  pixels  and  one  neighboring 
pixel  on  the  left  and  the  right  sides  of  thesaturated 
pixels  are  converted  to  zero  brightness  level  when 
sequential  saturated  pixels  are  detected.  Next,  all 
pixels  are  vertically  scanned  and  processed  in  the 
same  way.  Finally,  the  remaining  saturated  pixels 
are  converted  to  zero  level  (Fig.  2(c)). 

Step  3.  Differentiation  of  even  image  from  odd  im¬ 
age.  The  difference  image,  as  shown  in  Fig.  2(d), 
was  made  by  subtracting  the  even  image  (Fig.  2(b)) 
from  the  odd  image  (Fig.  2(c)).  At  the  same  time,  the 
brightness  histogram  of  the  difference  image  (Fig.  1 
(c))  is  made. 

Step  4.  Determination  of  a  temporary  threshold  for 


pupil  detection  in  the  difference  image.  The 

histogram  for  the  difference  image  was  scanned 
while  adding  the  pixel  number  in  each  bin  from  -255 
toward  255.  The  brightness  level  when  the  sum  of  the 
numbers  first  exceeds  half  of  the  total  (600  pixels)  is 
determined  as  the  background  brightness  (Bob  in  Fig. 
1(c)).  Next,  while  further  incrementing  the 
brightness  level  from  Bdb,  the  level  when  the  pixel 
number  first  becomes  zero  is  defined  as  the  upper 
limit  of  the  background  brightness  ( Bdbu  in  Fig.  1(c)). 
Step  5.  Solitary  island  elimination  in  the  difference 
image.  There  is  a  possibility  that  the  wrecks  of  the 
GRL  image  brighter  than  the  Bdbu  remain  as  solitary 
islands  as  shown  in  Fig.  2(d).  If  this  step  is  not  ap¬ 
plied,  the  pupil  image  binarized  by  the  Bdbu  becomes 
as  shown  in  Fig.  2(g). 

The  pupil  image  consists  of  a  relatively  large 
cluster  of  pixels.  I  n  contrast,  each  wreck  oftheGRL 
image  generally  consists  of  a  few  pixels.  So,  all  pixels 
in  the  difference  image  are  horizontally  scanned. 
When  the  brightness  levels  of  morethan  one  of  the 
four  surrounding  pixels  (the  upper,  lower,  right,  and 
left  pixels)  are  brighter  than  Bdbu  or  as  bright  as  - 
256,  thefocused  pixel  is  judged  to  be  a  part  of  a  soli¬ 
tary  island  and  its  brightness  level  is  replaced  by  - 
256.  Fig.  2(f)  shows  the  image  after  binarizing  the 
obtained  image  using  Bdbu. 

Step  6.  Determination  of  the  pixel  number  center  of 
gravity  (PNC)  of  the  pupil  in  the  difference  image. 
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By  horizontally  scanning  the  difference  image  (Fig.  2 
(e))  from  the  top  to  the  bottom  while  counting  the 
pixels  brighter  than  Bdbu  (high  pixel,  HP),  thetotal  of 
the  HP  is  determined  as  thetotal  high  pixel  number 
(THPN).  Next,  the  image  is  horizontally  scanned 
again  in  the  same  way  while  counting  the  H  P.  A  y 
coordinate  when  thetotal  of  the  HP  first  exceeds  half 
of  the  THPN  is  determined  as  that  of  the  PNC  of  the 
pupil  image.  Finally,  while  vertically  counting  the 
HP  from  the  I  eft  to  right,  a  x  coordinate  of  the  PNC  is 
determined  in  the  same  way. 

Step  7.  Determination  of  some  thresholds  using  im¬ 
ages  in  first  temporary  window  (TW).  T o  perfectly 
eliminatethe  GRL  and  other  noise  in  the  back¬ 
ground,  a  circular  TW  whose  center  is  the  PNC,  hav¬ 
ing  a  radius  of  5  mm  of  actual  size,  is  applied  (circles 
in  Fig.  3).  The  brightness  histograms  for  the  differ¬ 
ence  images  are  made  within  theTW  (Fig.  3(c)).  At 
thesametime,  within  theTW  in  the  difference  im¬ 
age,  the  pixels  brighter  than  Bdbu  and  those  not 
brighter  than  Bdbu  are  counted  as  the  pupil  and 
background  pixel  numbers  (A/p  and  A/s),  respectively. 

The  pixel  numbers  of  the  histogram  for  the  dif¬ 
ference  i  mage  are  added  from  -255  toward  255  i  n  the 
brightness  level.  The  brightness  level  when  the  sum 
exceeds  half  of  the  A/s  is  determined  as  the  back¬ 
ground  brightness  for  the  difference  image  within  the 
TW  ( Bdbw ).  Next,  the  pixel  numbers  of  the  same  his¬ 
togram  are  added  from  255  toward  -255  in  the 
brightness  level.  The  brightness  level  when  the  sum 
exceeds  half  of  the  A/p  is  determined  as  the  pupil 
brightness  ( Bdpw ).  A  threshold  for  pupil  detection 
(Thp)  is  determined  using  the  foil  owing  equation: 

Thp  HBdbw-^Bdpw)/2  (1) 

Step  8.  Remaking  of  the  difference  image  in  theTW. 
The  reduced  difference  image  within  the  TW  is 
remade  using  the  odd  image  (Fig.  2(c))  and  the  even 
image  (Fig.  2(b)). 

Step  9.  Determination  of  the  pupil  pixel  number 
(PPN)  and  the  PNC  of  the  pupil  pixels  in  theTW.  By 


scanning  only  the  pixels  in  theTW,  the  PNC  of  the 
pixels  having  the  brightness  levels  higher  than  Thp  is 
determined  as  the  PNC  of  the  pupil  pixels  by  the 
same  procedure  as  in  Step  6.  In  this  step,  the  number 
of  the  pupil  pixels  (PPN)  is  also  counted. 

Step  10.  Setting  an  appropriate  sized  window  around 
the  pupil  image.  TheTW  having  a  radius  of  5  mm  is 
usually  too  big  compared  to  the  actual  pupil  image 
size.  Accordingly,  a  window  as  small  as  possible 
should  be  applied  around  the  pupil  image.  First,  a 
TW  having  a  diameter  of  1  mm  of  actual  size  is 
applied  to  the  PNC  of  the  pupil  pixels  obtained  in 
Step  9.  The  TW  size  is  increased  gradually  while 
adjusting  theTW  center  totheupdated  PNC.  In  the 
final  TW,  the  maximum  of  the  distances  between 
each  of  the  pupil  pixels  and  thefinal  PNC  is  obtained. 
The  maximum  distance  +1  and  thefinal  PNC  are 
determined  as  the  radius  and  the  center  of  thefinal 
window,  respectively. 

Step  11.  Converting  the  final  window  to  the  original 
image  scale.  The  center  coordinates  and  radius  of 
the  final  window  are  converted  from  the  reduced 
scale  to  the  ordinal  image  scale. 

III.  Results 

Fig.  4  shows  the  results  of  windowing  the  pupil 
images  with  the  GRL,  eyeglasses  frame,  and  the 
other  backgrounds.  These  images  are  thetypical  ones 
obtained  in  the  foil  owing  experiment.  The  circle  and 
the  intersection  of  the  bars  indicatethe  window  ap¬ 
plied  and  its  center  coordinates,  respectively.  The 
appropriate  sized  windows  were  applied  to  each  ap¬ 
propriate  position. 

I  n  the  real  time  experiment,  a  computer  display 
was  placed  approximately  65  cm  facing  a  subject.  The 
screen  of  the  display  was  brightened  by  another  per¬ 
sonal  computer  to  mimic  an  actual  use  situation.  On 
the  screen  nine  small  markers  were  arranged  equally 
(3  by  3).  The  subject  was  asked  to  look  at  each  of  the 


Fig.  3.  Sample  images  in  each  step  of  the  windowing  process. 
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(a)  (b) 


Fig.  4.  Results  of  windowing  process  in  static  experiment. 


Fig.  5.  Results  of  windowing  process  in  real  time  experiment. 


nine  markers  successively  (every  3  seconds).  I  n  Fig.  5, 
thetop  and  middle  panel  indicate  the  coordinate  of 
the  extremes  of  the  window.  The  bottom  panel  indi¬ 
cates  the  PPN .  The  nonzero  PPN  and  the  feasible 
movements  of  the  center  and  extremes  of  the  wi  ndow 
indicate  that  the  window  was  consistently  and  relia¬ 
bly  applied  to  the  pupil  image  appropriately. 

IV.  Discussion 

When  the  eye  is  directed  to  a  point  far  from  the 
camera  axis,  thesclera  easily  brightens.  This  disturbs 
a  wide  range  of  detectors  of  the  line  of  sight.  However, 
we  confirmed  that  this  windowing  process  eliminates 
this  problem. 

We  have  already  begun  a  study  of  precise  pupil 


center  detection  within  the  window.  We  know  empiri¬ 
cally  that  the  threshold  determined  for  the  windowing 
process  ( Thp )  is  also  very  suitablefor  the  threshold  for 
binarizing  the  pupil  image  for  the  detailed  analysis 
inside  the  window. 

In  the  recursive  temporary  windowing  processing, 
theTW  was  gradually  increased  from  a  small  area  in 
each  frame  of  image.  This  processing  is  extremely 
effective  for  appropriate  windowing  in  the  case  that  a 
source  of  noise,  relatively  large  but  not  larger  than 
the  pupil  image,  exists  beside  the  pupil.  Such  a  case 
may  occur  in  the  case  where  the  GRL  does  not  satu¬ 
rate  in  the  odd  image.  This  size-increasing  method  is 
a  I  so  effective  for  shortening  the  calculation  time  com¬ 
pared  to  a  size-decreasing  method  [8]  because  it  takes 
less  time  for  calculation  in  smaller  sized  TWs. 

V.  Conclusions 

We  improved  the  pupil  detection  method  using 
two  light  sources  and  the  image  difference  method  we 
previously  proposed  as  follows:  (1)  For  convenience 
and  low  cost,  a  conventional  image  grabber  and  a 
personal  computer  were  used  for  image  analysis  in¬ 
stead  of  the  special  hardware  construction  in  the  pre¬ 
vious  study  [1],  (2)  A  circular  window  of  minimum 
dimensions  could  be  applied  around  the  pupil  image 
reliably  in  real  time.  (3)  Fully  automated  thresholding 
for  pupil  detection  was  realized.  (4)  All  the  processes 
listed  above  worked  well  on  subjects  wearing  eye¬ 
glasses.  (5)  The  processing  time  for  an  image  includ¬ 
ing  a  typical  sized  pupil  and  the  glass  reflections  was 
only  7.2  ms  on  the  Pentium  1 1  266  M  Hz-based  com¬ 
puter. 
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